3D-2D vision system for robotic carton unloading

ABSTRACT

Robotic carton loader or unloader incorporates three-dimensional (3D) and two-dimensional (2D) sensors to detect respectively a 3D point cloud and a 2D image of a carton pile within transportation carrier such as a truck trailer or shipping container. Edge detection is performed using the 3D point cloud, discarding segments that are two small to be part of a product such as a carton. Segments that are too large to correspond to a carton are 2D image processed to detect additional edges. Results from 3D and 2D edge detection are converted in a calibrated 3D space of the material carton loader or unloader to perform one of loading or unloading of the transportation carrier. Image processing can also detect jamming of products sequence from individually controllable zones of a conveyor of the robotic carton loader or unloader for singulated unloading.

PRIORITY CLAIM

This application is a continuation application of and claiming thebenefit of priority to U.S. application Ser. No. 15/481,969 entitled“3D-2D VISION SYSTEM FOR ROBOTIC CARTON UNLOADING” filed on Apr. 7,2017, which is a non-provisional application claiming the benefit ofpriority to Provisional Application No. 62/410,435 filed on Oct. 20,2016, Provisional Application No. 62/413,122, filed on Oct. 26, 2016,and Provisional Application No. 62/417,368, filed on Nov. 4, 2016 theentireties of which are incorporated by reference herein.

TECHNICAL FIELD

The present disclosure generally machine vision systems and moreparticularly to an autonomous vehicle that detects articles usingmachine vision in a material handling system.

BACKGROUND

Trucks and trailers loaded with cargo and products move across thecountry to deliver products to commercial loading and unloading docks atstores, warehouses, and distribution centers. Trucks can have a trailermounted on the truck, or can be of a tractor-semi trailer configuration.To lower overhead costs at retail stores, in-store product counts havebeen reduced, and products-in-transit now count as part of availablestore stock. Unloading trucks quickly at the unloading docks ofwarehouses and regional distribution centers has attained new prominenceas a way to refill depleted stock.

Trucks are typically loaded and unloaded with forklifts if the loads arepalletized and with manual labor if the products are stacked within thetrucks. Unloading large truck shipments manually with human laborers canbe physically difficult, and can be costly due to the time and laborinvolved. In addition, hot or cold conditions within a confined space ofa truck trailer or shipping container can be deemed unpleasant work.Consequently, a need exists for an improved unloading system that canunload bulk quantities of stacked cases and cargo from truck trailersmore quickly than human laborers and at a reduced cost.

In order to be economical, automation of loading or unloading needs tobe relatively fast. Generally-known approaches to unloading cartonsquickly have had extremely limited acceptance. It has been proposed tounload trailers without specific knowledge of exact locations of cartonswithin the trailer by applying bulk handling techniques. For example,the entire trailer can be tipped to move product toward a rear end door.For another example, cartons are placed on fabric layer that is pulledtoward the rear end door to dump the contents. In both instances,integrity of packaging and contents of cartons are jeopardized byapplying bulk handling techniques. At another extreme, it is known touse an articulated robotic arm with a machine vision sensor on an endeffector that scans a scene focused on an area of next position for apick or put. Extensive imaging processing and 3D point cloud processingoccurs in an attempt to detect cartons that are present within in anarrow portion of the carton pile. The wait time between operationsmakes it difficult to achieve an economical return on investment (ROI)for an automated trailer loader or unloader. Even with extensivedetection efforts, failures to detect individual cartons occur due todifficulty to sense edges that are too tightly aligned for 3D detectionor otherwise too optically camouflaged for 2D detection.

BRIEF SUMMARY

In one aspect, the present disclosure provides a method of determininglocations of individual cartons in a material handling system. In one ormore embodiments, the method includes receiving a two-dimensional (2D)image and a three-dimensional (3D) point cloud of at least one portionof a carton pile resting on a floor of a transportation carrier. Themethod includes detecting segments within the 3D point cloud. The methodincludes removing any segments that are smaller than a first threshold.The method includes determine whether any segments are less than asecond threshold. The method includes, in response to determining that aselected segment is less than the second threshold, qualifying theselected segment as a 3D detected carton. The method includes inresponse to determining that a selected segment is not less than thesecond threshold: (i) determining a 2D mask that corresponds to theselected segment; (ii) determining a portion of the 2D image thatcorresponds to the 2D mask; (iii) detecting segments within the portionof the 2D image; and (iv) qualifying detected segments as 2D detectedcartons. The method includes combining the 2D and 3D detected cartons ina detection result. The method includes converting the detection resultusing calibration information into 3D locations relative to a roboticcarton handling system for a selected one of a loading operation and anunloading operation.

In another aspect, the present disclosure provides a robotic cartonhandling system for unloading cartons in a carton pile. The roboticcarton handling system is movable across a floor. In one or moreembodiments, the robotic carton handling system includes a mobile body.The robotic carton handling system includes a movable roboticmanipulator attached to the mobile body. The movable robotic manipulatorincludes an end effector at an end thereof. The end effector unloads orloads one or more cartons from the carton pile. A conveyor mounted onthe mobile body receives the one or more cartons from the end effectorand to move the one or more cartons towards a rear of the robotic cartonhandling system. A carton detection system includes one or more sensorscoupled respectively to one of the mobile body and the movable roboticmanipulator. The one or more sensors provide a 2D image and a 3D pointcloud of at least one portion of a carton pile resting on a floor of atransportation carrier. A processing subsystem is in communication withthe one or more sensors. The processing subsystem detects segmentswithin the 3D point cloud. The processing subsystem removes any segmentsthat are smaller than a first threshold. The processing subsystemdetermines whether any segments are less than a second threshold. Theprocessing subsystem, in response to determining that a selected segmentis less than the second threshold, qualifies the selected segment as a3D detected carton. The processing subsystem, in response to determiningthat a selected segment is not less than the second threshold: (i)determines a 2D mask that corresponds to the selected segment; (ii)determines a portion of the 2D image that corresponds to the 2D mask;(iii) detects segments within the portion of the 2D image; and (iv)qualifies detected segments as 2D detected cartons. The processingsubsystem combines the 2D and 3D detected cartons in a detection result.The processing subsystem converts the detection result using calibrationinformation into 3D locations relative to a robotic carton handlingsystem for a selected one of a loading operation and an unloadingoperation.

In an additional aspect, the present disclosure provides a materialhandling system including a robotic carton handling system for unloadingcartons in a carton pile. The robotic carton handling system is movableacross a floor. In one or more embodiments, the robotic canon handlingsystem includes a mobile body and a movable robotic manipulator attachedto the mobile body. The movable robotic manipulator includes an endeffector at an end thereof. The end effector unloads one or more cartonsfrom the carton pile. A conveyor mounted on the mobile body receives theone or more cartons from the end effector and to move the one or morecartons towards a rear of the robotic carton handling system. A cartondetection system includes one or more sensors coupled respectively toone of the mobile body and the movable robotic manipulator. The one ormore sensors provide a 2D image and a 3D point cloud of at least oneportion of a carton pile resting on a floor of a transportation carrier.A processing subsystem is in communication with the one or more sensors.The processing subsystem detects segments within the 3D point cloud. Theprocessing subsystem removes any segments that are smaller than a firstthreshold. The processing subsystem determines whether any segments areless than a second threshold. The processing subsystem in response todetermining that a selected segment is less than the second threshold,qualifies the selected segment as a 3D detected carton. The processingsubsystem in response to determining that a selected segment is not lessthan the second threshold: (i) determines a 2D mask that corresponds tothe selected segment; (ii) determines a portion of the 2D image thatcorresponds to the 2D mask; (iii) detects segments within the portion ofthe 2D image; and (iv) qualifies detected segments as 2D detectedcartons. The processing subsystem combines the 2D and 3D detectedcartons in a detection result. The processing subsystem converts thedetection result using calibration information into 3D locationsrelative to a robotic carton handling system for a selected one of aloading operation and an unloading operation. An automation controlleris in communication with the processing subsystem. The automationcontroller causes the robotic carton manipulator to perform the selectedone of the loading operation and the unloading operation by the roboticcarton handling system using the 3D locations. An extendable conveyorsystem has a proximal end coupled to a stationary conveyor system. Theextendable conveyor has a movable distal end positioned proximate to therobotic carton handling system to transfer cartons between thestationary conveyor and the robotic carton handling system.

The above summary contains simplifications, generalizations andomissions of detail and is not intended as a comprehensive descriptionof the claimed subject matter but, rather, is intended to provide abrief overview of some of the functionality associated therewith. Othersystems, methods, functionality, features and advantages of the claimedsubject matter will be or will become apparent to one with skill in theart upon examination of the following figures and detailed writtendescription.

BRIEF DESCRIPTION OF THE DRAWINGS

The description of the illustrative embodiments can be read inconjunction with the accompanying figures. It will be appreciated thatfor simplicity and clarity of illustration, elements illustrated in thefigures have not necessarily been drawn to scale. For example, thedimensions of some of the elements are exaggerated relative to otherelements. Embodiments incorporating teachings of the present disclosureare shown and described with respect to the figures presented herein, inwhich:

FIG. 1 illustrates a side view with functional block diagram of arobotic carton handling system and extendable conveyor unloading cartonsfrom within a carton pile container using a vision system that ismulti-quadrant and combined two-dimensional (2D) and three-dimensional(3D), according to one or more embodiments;

FIG. 2 illustrates a top isometric view of the robotic carton handlingsystem of FIG. 1, according to one or more embodiments;

FIG. 3 illustrates a bottom isometric view of the robotic cartonhandling system of FIG. 1, according to one or more embodiments;

FIG. 4 illustrates a front side view of a forward portion of the roboticcarton handling system of FIG. 1, according to one or more embodiments;

FIG. 5 illustrates a right side view of a forward portion of the roboticcarton handling system of FIG. 1, according to one or more embodiments;

FIG. 6 illustrates an exemplary computing environment for an onboardunloading controller of the robotic carton handling system of FIG. 1,according to one or more embodiments;

FIG. 7 illustrates a vision system of the robotic carton handling systemof FIG. 1, according to one or more embodiments;

FIG. 8 illustrates a flow diagram of a method for carton detection of acarton pile resting on a floor of a transportation carrier, according toone or more embodiments;

FIG. 9 illustrates flow diagram of a an example method for 2D, 3D, and3D-guided 2D box detection, according to one or more embodiments;

FIG. 10 illustrates flow diagram of a another example method for 2D, 3D,and 3D-guided 2D box detection, according to one or more embodiments;

FIG. 11 illustrates flow diagram of a an example method for 3Dprocessing performed as part of the method of FIG. 10, according to oneor more embodiments;

FIG. 12 illustrates flow diagram of a an example method for 2Dprocessing performed as part of the method of FIG. 10, according to oneor more embodiments; and

FIG. 13 illustrates a simplified flow diagram of an example method for2D, 3D, and 3D-guided 2D box detection, according to one or moreembodiments.

DETAILED DESCRIPTION

Robotic carton loader or unloader incorporates three-dimensional (3D)and two-dimensional (2D) sensors to detect respectively a 3D point cloudand a 2D image of a carton pile within transportation carrier such as atruck trailer or shipping container. Cartons can be identified in partwithin the 3D point cloud by discarding segments that are two small tobe part of a product such as a carton. Segments that are too large tocorrespond to a carton are 2D image processed to detect additionaledges. Results from 3D and 2D edge detection are converted in acalibrated 3D space of the material carton loader or unloader to performone of loading or unloading of the transportation carrier.

In one aspect of the present disclosure, a customized Red-Green-Blue andDepth (RGB-D) vision solution is provided for autonomous truckunloaders. A RGB-D sensor system was designed using a combination ofindustrial Depth and RGB sensor, specifically tailored to the needs of atruck unloader. Four such units combined gives the RGB, depth and RGB-Ddata across the entire width and height of a trailer. Each of the RGBcameras has unique projection parameters. Using those and the relativeposition of the Depth and RGB sensor, the 3D from the depth sensor ismapped onto 2D image data from RGB sensor and vice versa. The data from3D and 2D RGB can be stitched together on the higher level to obtain anentire scene.

After commissioning the product, the lifetime is expected to be long.Average ambient operating temperature, source voltage consistency, shockand vibration isolation, isolation from high power emitters, ifcontrolled properly, will extend the life of the sensors and the system.At the end of life per component, more false positives and falsenegatives are expected. Dead points can be monitored on the sensor tothe point that when a minimum number of points is reached, the sensorcan be flagged for replacement. Component pieces are serviceableassuming original parts or compatible replacements can be sourced.

In the following detailed description of exemplary embodiments of thedisclosure, specific exemplary embodiments in which the disclosure maybe practiced are described in sufficient detail to enable those skilledin the art to practice the disclosed embodiments. For example, specificdetails such as specific method orders, structures, elements, andconnections have been presented herein. However, it is to be understoodthat the specific details presented need not be utilized to practiceembodiments of the present disclosure. It is also to be understood thatother embodiments may be utilized and that logical, architectural,programmatic, mechanical, electrical and other changes may be madewithout departing from general scope of the disclosure. The followingdetailed description is, therefore, not to be taken in a limiting sense,and the scope of the present disclosure is defined by the appendedclaims and equivalents thereof.

References within the specification to “one embodiment,” “anembodiment,” “embodiments”, or “one or more embodiments” are intended toindicate that a particular feature, structure, or characteristicdescribed in connection with the embodiment is included in at least oneembodiment of the present disclosure. The appearance of such phrases invarious places within the specification are not necessarily allreferring to the same embodiment, nor are separate or alternativeembodiments mutually exclusive of other embodiments. Further, variousfeatures are described which may be exhibited by some embodiments andnot by others. Similarly, various requirements are described which maybe requirements for some embodiments but not other embodiments.

It is understood that the use of specific component, device and/orparameter names and/or corresponding acronyms thereof, such as those ofthe executing utility, logic, and/or firmware described herein, are forexample only and not meant to imply any limitations on the describedembodiments. The embodiments may thus be described with differentnomenclature and/or terminology utilized to describe the components,devices, parameters, methods and/or functions herein, withoutlimitation. References to any specific protocol or proprietary name indescribing one or more elements, features or concepts of the embodimentsare provided solely as examples of one implementation, and suchreferences do not limit the extension of the claimed embodiments toembodiments in which different element, feature, protocol, or conceptnames are utilized. Thus, each term utilized herein is to be given itsbroadest interpretation given the context in which that terms isutilized.

FIG. 1 illustrates a robotic carton handling system 100 having amanipulator such as a robotic arm assembly 102 unloads cartons 104 froma carton pile 106 inside of a carton pile container 108, such as atrailer, shipping container, storage unit, etc. Robotic arm assembly 102places the cartons 104 onto a conveyor system 110 of the robotic cartonhandling system 100 that conveys the cartons 104 back to an extendableconveyor 112 that follows a mobile body 114 of the robotic cartonhandling system 100 into the carton pile container 108. The extendableconveyor 112 in turn conveys die cartons 104 to a material handlingsystem 116 such as in a warehouse, store, distribution center, etc.

In one or more embodiments, the robotic carton handling system 100autonomously unloads a carton pile 106 resting on a floor 118 of thecarton pile container 108. The mobile body 114 is self-propelled andmovable across the floor 118 from outside to the innermost portion ofthe carton pile container 108. Right and left lower arms 120 of therobotic arm assembly 102 are pivotally attached at a lower end 122respectively to the mobile body 114 on opposing lateral sides of theconveyor system 110 passing there between. The right and left lower arms120 rotate about a lower arm axis 124 that is perpendicular to alongitudinal axis 126 of the conveyor system 110. An upper arm assembly128 of the robotic arm assembly 102 has a rear end 130 pivotallyattached at an upper end 132 respectively of the right and left lowerarms 120 to pivotally rotate about an upper arm axis 134 that isperpendicular to the longitudinal axis 126 of the conveyor system 110and parallel to the lower arm axis 124. A manipulator head 136 isattached to a front end 138 of the upper arm assembly 128 and engages atleast one carton 104 at a time from the carton pile 106 resting on thefloor 118 for movement to the conveyor system 110. The pivotal andsimultaneous mirrored movement of the right and left lower arms 120maintains the upper arm axis 134 at a relative height above the conveyorsystem 110 that enables the at least one carton 104 to be conveyed bythe conveyor system 110 without being impeded by the robotic armassembly 102 as soon as the manipulator head 136 is clear. In one ormore embodiments, the robotic carton handling system 100 includes a lift140 attached between the mobile body 114 and a front portion 142 of theconveyor system 110. The lift 140 moves the front portion 142 of theconveyor system 110 relative to the floor 118 to reduce spacingunderneath the at least one carton 104 during movement from the cartonpile 106 to the conveyor system 110.

A higher level system can assign an autonomous robotic vehiclecontroller 144 of the robotic carton handling system 100 to a particularcarton pile container 108 and can receive information regarding progressof loading/unloading as well as provide a channel for telecontrol. Ahuman operator could selectively intervene when confronted with an errorin loading or unloading. The higher level system can include a hostsystem 146 that handles external order transactions that are to becarried out by the material handling system 116. Alternatively or inaddition, a warehouse execution system (WES) 148 can provide verticalintegration of a warehouse management system (WMS) 150 that performsorder fulfillment, labor management, and inventory tracking for afacility 152 such as a distribution center. WES 148 can include avertically integrated warehouse control system (WCS) 154 that controlsautomation that carries out the order fulfillment and inventorymovements requested by the WMS 150.

In one or more embodiments, once assigned by the WES 148 or manuallyenabled, the robotic carton handling system 100 can operate autonomouslyunder control of a robotic vehicle controller 154 in: (i) moving into acarton pile container 108, (ii) performing one of loading or unloadingthe carton pile container 108, and (iii) moving out of the carton pilecontainer 108. In order to navigate within the carton pile container 108and to expeditiously handle cartons 104 therein, a carton detectionsystem 166 of the robotic vehicle controller 154 includes sensors 157attached respectively to one of the mobile body 114 and the movablerobotic manipulator (robotic arm assembly 102) to provide atwo-dimensional (2D) image and a three-dimensional (3D) point cloud ofat least one portion of the carton pile 106 resting on a floor 159 of acarton pile container 108. The carton pile container 108 can bestationery or mobile, such as transportation carriers for highway,railway or shipping on navigable waters.

Controller 144 provides an exemplary environment within which one ormore of the described features of the various embodiments of thedisclosure can be implemented. A controller 144 can be implemented as aunitary device or distributed processing system. The controller 144includes functional components that communicate across a systeminterconnect of one or more conductors or fiber optic fabric that forclarity is depicted as a system bus 156. System bus 156 may include adata bus, address bus, and control bus for communicating data, addressesand control information between any of these coupled units. Functionalcomponents of the controller 144 can include a processor subsystem 158consisting of one or more central processing units (CPUs), digitalsignal processor/s (DSPs) and processor memory. Processor subsystem 158may include any instrumentality or aggregate of instrumentalitiesoperable to compute, classify, process, transmit, receive, retrieve,originate, switch, store, display, manifest, detect, record, reproduce,handle, or utilize any form of information, intelligence, or data forbusiness, scientific, control, or other purposes including control ofautomation equipment of a material handling system.

In accordance with various aspects of the disclosure, an element, or anyportion of an element, or any combination of elements may be implementedwith processor subsystem 158 that includes one or more physical devicescomprising processors. Non-limiting examples of processors includemicroprocessors, microcontrollers, digital signal processors (DSPs),field programmable gate arrays (FPGAs), programmable logic devices(PLDs), programmable logic controllers (PLCs), state machines, gatedlogic, discrete hardware circuits, and other suitable hardwareconfigured to perform the various functionality described throughoutthis disclosure. One or more processors in the processing system mayexecute instructions. A processing system that executes instructions toeffect a result is a processing system which is configured to performtasks causing the result, such as by providing instructions to one ormore components of the processing system which would cause thosecomponents to perform acts which, either on their own or in combinationwith other acts performed by other components of the processing systemwould cause the result.

Controller 144 may include a network interface (I/F) device 160 thatenables controller 144 to communicate or interface with other devices,services, and components that are located external to controller 144,such as WES 148. These networked devices, services, and components caninterface with controller 144 via an external network, such as examplenetwork 162, using one or more communication protocols. Network 162 canbe a local area network, wide area network, personal area network, andthe like, and the connection to and/or between network and controller144 can be wired or wireless or a combination thereof. For purposes ofdiscussion, network 162 is indicated as a single collective componentfor simplicity. However, it is appreciated that network 162 can compriseone or more direct connections to other devices as well as a morecomplex set of interconnections as can exist within a wide area network,such as the Internet or on a private intranet. For example, aprogramming workstation 1924 can remotely modify programming orparameter settings of controller 144 over the network 162. Various linksin the network 162 can wired or wireless. Controller 144 can communicatevia a device interface 168 with a number of on-board devices such aslights, indicators, manual controls, etc. Device interface 168 caninclude wireless links and wired links. For example, the controller 144can direct the extendable conveyor 112 follow the robotic cartonhandling system 100 into the carton pile container 108 or to lead therobotic carton handling system 100 out of the carton pile container 108.

Controller 144 can include several distributed subsystems that manageparticular functions of the robotic carton handling system 100. Anautomation controller 170 can receive location and spatial calibrationinformation from the 3D/2D carton detection system 166 and use this datato coordinate movement of the mobile body 114 via a vehicle interface172 and movement by payload components such as robotic arm assembly 102and the lift 140 that moves the front portion 142 of the conveyor system110.

The 3D/2D carton detection system 166 can include depth sensing usingbinocular principles, lidar principles, radar principles, or sonarprinciples. To avoid dependency on consistent ambient lightingconditions, an illuminator 169 can provide a consistent or adjustableamount of illumination in one or more spectrum bandwidths such as visuallight or infrared. The illumination can be narrowly defined in thevisual spectrum enabling filtration of most of the ambient light.Alternatively, the illumination can be outside of the visual range suchthat the illumination is not distracting to human operators. The 3D/2Dcarton detection system 166 can receive 2D and 3D sensor data from frontRGB-D sensors 176 that view an interior of the carton pile container 108and the carton pile 106. For these and other purposes, the 3D/2D cartondetection system 166 can include various applications or components thatperform processes described later in the present application. Forexample, the 3D/2D carton detection system 166 can include a 2D processmodule 180, a 3D process module 182, and a 3D-guided 2D process module184.

System memory 164 can be used by processor subsystem 158 for holdingfunctional components such as data and software such as a 3D/2D cartondetection system 166. Software may be construed broadly to meaninstructions, instruction sets, code, code segments, program code,programs, subprograms, software modules, applications, softwareapplications, software packages, routines, subroutines, objects,executables, threads of execution, procedures, functions, etc., whetherreferred to as software, firmware, middleware, microcode, hardwaredescription language, function block diagram (FBD), ladder diagram (LD),structured text (ST), instruction list (IL), and sequential functionchart (SFC) or otherwise. The software may reside on a computer-readablemedium.

For clarity, system memory 164 can include both random access memory,which may or may not be volatile, nonvolatile data storage. Systemmemory 164 contain one or more types of computer-readable medium, whichcan be a non-transitory or transitory. Computer-readable mediumincludes, by way of example, a magnetic storage device (e.g., hard disk,floppy disk, magnetic strip), an optical disk (e.g., compact disk (CD),digital versatile disk (DVD)), a smart card, a flash memory device(e.g., card, stick, key drive), random access memory (RAM), read onlymemory (ROM), programmable ROM (PROM), erasable PROM (EPROM),electrically erasable PROM (EEPROM), a register, a removable disk, andany other suitable medium for storing software and/or instructions thatmay be accessed and read by a computer. The computer-readable medium maybe resident in the processing system, external to the processing system,or distributed across multiple entities including the processing system.The computer-readable medium may be embodied in a computer-programproduct. By way of example, a computer-program product may include acomputer-readable medium in packaging materials. Those skilled in theart will recognize how best to implement the described functionalitypresented throughout this disclosure depending on the particularapplication and the overall design constraints imposed on the overallsystem.

FIG. 2 illustrates that the upper arm assembly 128 of the robotic cartonhandling system 100 includes a rotatable gantry 201 having the rear end130 pivotally attached at the upper arm axis 134 to the left and rightlower arms 120. The rotatable gantry 201 has a lateral guide 203 at anextended end 205. The upper arm assembly 128 includes an end arm 207proximally attached for lateral movement to the lateral guide 203 of therotatable gantry 201 and distally attached to the manipulator head 136.The end arm 207 laterally translates to reach an increased lateral area.Thereby a lighter weight and more maneuverable manipulator head 136 canbe employed. FIGS. 2-5 illustrate that an equipment cabinet 209 archesover a back portion of the conveyor system 110. With particularreference to FIG. 5, clearance under the equipment cabinet 209 defines ajam height 210 that can be determined based upon sensor data from rear3D/2D sensors 178 mounted on the equipment cabinet 209 for any cartonsreceived on the front portion 142 of the conveyor system 110. Forexample, the rear 3D/2D sensors 178 can include a 2D infrared sensor211, a 3D depth sensor 213, and a 2D optical sensor 215. Front 3D/2Dsensors 176 can include spatially separated sensors that operate indifferent spectrum and dimensions in order to detect articles such asproduct, cartons, boxes, cases, totes, etc., (cartons 104) under anumber of stacking arrangements, lighting conditions, etc. Mountingsensors on the end effector (manipulator head 136) also allows varying avantage point, such as looking downward onto the carton pile 106 tobetter differentiate top-most cartons 104.

With particular reference to FIGS. 2 and 4, in an exemplary embodimentthe front 3D/2D sensors 176 include a top left 2D sensor 217, a top left3D sensor 219, a top right 2D sensor 221, and a top right 3D sensor 223on the manipulator head 136. The front 3D/2D sensors 176 include bottomleft 2D sensor 227, a bottom left 3D sensor 229, a bottom right 2Dsensor 231, and a bottom right 3D sensor 233 on the front end of themobile body 114.

FIG. 6 illustrates exemplary components of a material handling system600 that includes robotic carton handling system 601 suitable for use invarious embodiments. The robotic carton handling system 601 may includean external monitor 602, a network interface module 604, an HMI module606, an input/output module (I/O module 608), a robotic arm and aconveyor system 615 that includes a drives/safety module 612 and amotion module 614, a programmable logic controller (or PLC 618), a basemotion module 620 that includes a vehicle controller module 622 and amanual control module 624, and a vision system 626 (or visualizationsystem) that may include one or more personal computing devices 628 (or“PCs”) and sensor devices 630. In some embodiments, vision system 626 ofthe robotic carton handling system 601 may include a PC 628 connected toeach sensor device 630. In embodiments in which more than one sensordevice 630 is present on the robotic carton handling system 601, the PCs628 for each sensor device 630 may be networked together and one of thePCs 628 may operate as a master PC 628 receiving data from the otherconnected PCs 628, may perform data processing on the received data andits own data (e.g., coordinate transformation, duplicate elimination,error checking, etc.), and may output the combined and processed datafrom all the PCs 628 to the PLC 618. In some embodiments, the networkinterface module 604 may not have a PLC inline between itself and the PC628, and the PLC 618 may serve as the Vehicle Controller and/orDrives/Safety system. Sensor devices 630 can include 2D image capturingdevices (ICDs) 631 and 3D image capturing devices (ICDs) 633 segregatedinto sectors for different viewing portions or vantage points. Subsetscan include rear mounted sensors 635, end effector mounted sensors 637,and vehicle mounted sensors 639.

The robotic carton handling system 601 may connect to remote locationsor systems with the network interface module 604 (e.g., a Wi-Fi™ radio,etc.) via a network 603, such as a local area Wi-Fi™ network. Inparticular, the network interface module 604 may enable the roboticcarton handling system 601 to connect to an external monitor 602. Theexternal monitor 602 may be anyone of a remote warehouse or distributioncenter control room, a handheld controller, or a computer, and mayprovide passive remote viewing through the vision system 626 of therobotic carton handling system 601. Alternately, the external monitor602 may override the programming inherent in the vision system 626 andassume active command and control of the robotic carton handling system601. Programming for the robotic carton handling system 601 may also becommunicated, operated and debugged through external systems, such asthe external monitor 602. Examples of an external monitor 602 thatassumes command and control may include a remotely located humanoperator or a remote system, such as a warehouse or distribution serversystem (i.e., remote device as described above). Exemplary embodimentsof using an external monitor 602 to assume command and control of therobotic carton handling system 601 may include human or computerintervention in moving the robotic carton handling system 601, such asfrom one unloading bay to another, or having the external monitor 602assume control of the robotic arm to remove an item (e.g., box, carton,etc.) that is difficult to unload with autonomous routines. The externalmonitor 602 may include any of: a visual monitor, a keyboard, ajoystick, an I/O port, a CD reader, a computer, a server, a handheldprogramming device, or any other device that may be used to perform anypart of the above described embodiments.

The robotic carton handling system 601 may include a human machineinterface module 606 (or HMI module 606) that may be used to controland/or receive output information for the robot arm and conveyor system615 and/or the base motion module 620. The HMI module 606 may be used tocontrol (or may itself include) a joystick, a display, and a keypad thatmay be used for re-programming, over-riding the autonomous control ofthe machine, and driving the robotic carton handling system 601 frompoint to point. Actuators 609 may be actuated individually or in anycombination by the vision system 626 via the I/O module 608, anddistance sensors 610 may be used to assist in guiding the robotic cartonhandling system 601 into an unloaded area (e.g., a trailer). The I/Omodule 608 may connect the actuators 609 and distance sensors 610 to thePLC 618. The robotic arm and conveyor system 615 may include allcomponents needed to move the arm and/or the conveyor, such asdrives/engines and motion protocols or controls. The base motion module620 may be the components for moving the entirety of the robotic cartonhandling system 601. In other words, the base motion module 620 may bethe components needed to steer the vehicle into and out of unloadingareas.

The PLC 618 that may control the overall electromechanical movements ofthe robotic carton handling system 601 or control exemplary functions,such as controlling the robotic arm or a conveyor system 615. Forexample, the PLC 618 may move the manipulator head of the robotic arminto position for obtaining items (e.g., boxes, cartons, etc.) from awall of items. As another example, the PLC 618 may control theactivation, speed, and direction of rotation of kick rollers, and/orvarious adjustments of a support mechanism configured to move afront-end shelf conveyor, such as front portion 142 of conveyor system110 (FIG. 1). The PLC 618 and other electronic elements of the visionsystem 626 may mount in an electronics box (not shown) located under aconveyor, adjacent to a conveyor, or elsewhere on the robotic cartonhandling system 601. The PLC 618 may operate all or part of the roboticcarton handling system 601 autonomously and may receive positionalinformation from the distance sensors (not shown). The I/O module 608may connect the actuators and the distance sensors 610 to the PLC 618.

The robotic carton handling system 601 may include a vision system 626that comprises sensor devices 630 (e.g., cameras, 3D sensors, etc.) andone or more computing device 628 (referred to as a personal computer or“PC” 628). The robotic carton handling system 601 may use the sensordevices 630 and the one or more PC 628 of the vision system 626 to scanin front of the robotic carton handling system 601 in real time or nearreal time. The forward scanning may be triggered by the PLC 618 inresponse to determining the robotic carton handling system 601, such asa trigger sent in response to the robotic carton handling system 601being in position to begin detecting cartons in an unloading area. Theforward scanning capabilities may be used for collision avoidance, sentto the human shape recognition (safety), sizing unloaded area (e.g., thetruck or trailer), and for scanning the floor of the unloaded area forloose items (e.g., cartons, boxes, etc.). The 3D capabilities of thevision system 626 may also provide depth perception, edge recognition,and may create a 3D image of a wall of items (or carton pile). Thevision system 626 may operate alone or in concert with the PLC 618 torecognize edges, shapes, and the near/far distances of articles in frontof the robotic carton handling system 601. For example the edges anddistances of each separate carton in the wall of items may be measuredand calculated relative to the robotic carton handling system 601, andvision system 626 may operate alone or in concert with the PLC 618 tomay select specific cartons for removal.

In some embodiments, the vision system 626 may provide the PLC withinformation such as: specific XYZ coordinate locations of cartonstargeted for removal from the unloading area, and one or more movementpaths for the robotic arm or the mobile body of the robotic cartonhandling system 601 to travel. The PLC 618 and the vision system 626 maywork independently or together such as an iterative move and visualcheck process for carton visualization, initial homing, and motionaccuracy checks. The same process may be used during vehicle movement,or during carton removal as an accuracy check. Alternatively, the PLC618 may use the move and visualize process as a check to see whether oneor more cartons have fallen from the carton pile or repositioned sincethe last visual check. While various computing devices and/or processorsin FIG. 6, such as the PLC 618, vehicle controller module 622, and PC628, have been described separately, in the various embodimentsdiscussed in relation to FIG. 6 and all the other embodiments describedherein, the described computing devices and/or processors may becombined and the operations described herein performed by separatecomputing devices and/or processors may be performed by less computingdevices and/or processors, such as a single computing device orprocessor with different modules performing the operations describedherein. As examples, different processors combined on a single circuitboard may perform the operations described herein attributed todifferent computing devices and/or processors, a single processorrunning multiple threads/modules may perform operations described hereinattributed to different computing devices and/or processors, etc.

An extendable conveyor system 632 can convey articles from the roboticcarton handling system 601 to other portions of a material handlingsystem 600. As the robotic carton handling system 601 advances orretreats, a vision device 634 on one or the extendable conveyor system632 and robotic carton handling system 601 can image a target 636 on theother. Vision system 626 can perform image processing to detect changesin size, orientation and location of the target 636 within the field ofview of the vision device 636. Device interfaces 638, 640 respectivelyof the extendable conveyor system 632 and the robotic carton handlingsystem 601 can convey vision information or movement commands. Forexample, PLC 618 can command an extension motion actuator 642 on theextendable conveyor system 632 to correspond to movements of the roboticcarton handling system 601 to keep the extendable conveyor system 632and the robotic carton handling system 601 in alignment and in properspacing. In one embodiment, the device interfaces 638, 640 utilize ashort range wireless communication protocol such as a Personal AccessNetwork (PAN) protocol. Examples of PAN protocols which may be used inthe various embodiments include Bluetooth®, IEEE 802.15.4, and Zigbee®wireless communication protocols and standards.

FIG. 7 illustrates a data flow within an example vision system 700 ofthe robotic carton handling system 100 (FIG. 1). An end effector 702includes a first RGD-D unit 704 and a second RGD-D unit 706. A vehicle708 includes a rear IR-RGB-D unit 710 positioned for conveyor screeningand including an RGB unit 712, a 3D unit 714, and an IR unit 716. Thevehicle 708 includes a third RGB-D unit 718 and a fourth RGB-D unit 720,each having an RGB sensor 722 and a depth sensor 724. A first PC 726 isin communication with the first and second RGD-D units 704, 706 and witha PLC 728 that performs automation control of the robotic cartonhandling system 100 (FIG. 1). A second PC 730 is in communication withthe third RGB-D unit 718 and PLC 728. A third PC 732 is in communicationwith the fourth RGB-D unit 720 and PLC 728. A fourth PC 734 is incommunication with the fourth RGB-D unit 720 and PLC 728. The first,second, third and fourth PCs 726, 730, 732, 734 each include 2D processmodule 736 and a 3D process module 738. PLC 728 sends a trigger signal740 to each of the first, second, third and fourth PCs 726, 730, 732,734. Each of the first, second, third and fourth PCs 726, 730, 732, 734in turn send a trigger signal 742 for data to respective assigned first,second, third and fourth RGB-D units 704, 706, 718, 720 and rearIR-RGB-D unit 710. The first, second, third and fourth RGB-D units 704,706, 718, 720 respond with RGB-D data 744. The rear IR-RGB-D unit 710responds with IR-RGB-D data 746. The first, second, and third PCs 726,730, 732 analyze RGB-D data 744 and provide a Cartesian 3D array 748 ofbox locations for assigned sectors or quadrants. Fourth PC 734 analyzesthe IR-RGB-D data 746 and produces zone presence and height data 752.PLC 728 can consolidate this data or one of the first, second, third andfourth PCs 726, 730, 732, 734 can perform this role for the PLC 728.

FIG. 8 illustrates a method 800 of determining locations of individualcartons in a material handling system. In one or more embodiment, themethod 800 includes receiving a 2D image and a 3D point cloud from oneor more sensors positioned on the robotic carton handling system todetect the one portion of the carton pile resting on a floor of atransportation carrier (block 802). Method 800 includes receiving a 2Dimage and 3D point cloud from another one or more sensors positioned onthe robotic carton handling system to detect a contiguous portion of thecarton pile (block 804). Method 800 includes detecting segments withinthe 3D point cloud (block 806). Method 800 includes removing anysegments that are smaller than a first threshold (block 808). Method 800includes determining whether any segments are less than a secondthreshold that is larger than the first threshold (decision block 810).In response to determining that a selected segment is less than thesecond threshold in decision block 810, qualifying the selected segmentas a 3D detected carton (block 812). Additional rules can be imposedother than a size expected of a carton such as requiring edges to beapproximately vertical or horizontal.

In response to determining that a selected segment is not less than thesecond threshold in decision block 810, method 800 includes:

(i) determining a 2D mask that corresponds to the selected segment(block 814);

(ii) determining a portion of the 2D image that corresponds to the 2Dmask (block 816);

(iii) detecting segments within the portion of the 2D image (block 818);and

(iv) qualifying detected segments as 2D detected cartons at least inpart by discarding edges that form a smaller rectangle fully encompassedwithin a larger rectangle (block 820).

After 3D detection of block 812 and any 3D guided 2D detection of block820, method 800 includes combining the 2D and 3D detected cartons in adetection result for each portion of the scene (block 822). Method 800includes converting the detection result using calibration informationinto 3D locations relative to a robotic carton handling system for aselected one of a loading operation and an unloading operation (block824). Method 800 includes combining the 2D and 3D detected cartons fromboth the one portion and the contiguous portion of the scene to form thedetection result (block 826). Method 800 includes performing theselected one of the loading operation and the unloading operation by therobotic carton handling system using the 3D locations (block 828).

FIG. 9 illustrates an example method 900 for 2D, 3D, and 3D-guided 2Dbox detection. According to one or more embodiments, the method 900includes obtaining optical red-green-blue and depth (RGB-D) sensor data(block 902). Method 900 includes a 2D box detection sub-process 904, a3D box detection sub-process 906, and a 3D-guided 2D box detectionsub-process 908 that analyze the RGB-D sensor data. Each type ofdetection either singularly or in combination can find edges of cartons,boxes, articles, etc., as well as localizing the robotic carton handlingsystem within a carton pile carrier such as a truck trailer or shippingcontainer.

Beginning with 2D box detection sub-process 904, the method 900 includespreparing 2D RGB input from the RGB-D sensor data (block 910). Method900 includes performing 2D edge detection to create an edge map (block912). Method 900 includes detecting contours of the 2D edge map (block914). Method 900 includes filtering out contours that are less than athreshold (block 916). For example, cartons can have printing or labelsthat could be deemed to be generally rectangular. Method 900 includescompiling total box detection from scene derived by 2D processing as 2Dresult (block 918). An output from 2D box detection sub-process 904 is2D box detection data structure 920.

Continuing with 3D box detection sub-process 906, method 900 includespreparing a 3D point cloud (block 922). Method 900 includespreprocessing 3D point cloud for noise removal (block 924). For example,a resolution level of the 3D point cloud can be reduced to spatiallyfilter and smooth the 3D point cloud. Method 900 includes extracting 3Dsegments using cluster-based segmentation method for 3D box detection(block 926). Method 900 includes determining whether each clusterqualifies as a box (decision block 928). For example, segments can befiltered for forming rectangles within a certain range of sizes andwithin a certain range of aspect ratios. In response to determining thatcertain clusters do qualify as a box in decision block 928, method 900includes compiling a total box detection from 3D point cloud derived by3D processing as 3D result (block 930). The 3D box detection sub-process906 outputs at least in part a 3D box detection data structure 932. Inresponse to determining that a cluster does not qualify as a box indecision block 928, method 900 includes creating a 2D mask region fromthe unqualified cluster/s (block 934). In which case, the 3D boxdetection sub-process 906 outputs at least in part a 2D mask region datastructure 936.

The 3D-guided 2D box detection sub-process 908 receives the 2D boxdetection data structure 920 from 2D box detection sub-process 904 andthe mask region data structure 932 from the 2D box detection sub-process906. Method 900 includes filtering 2D box detection data structure 920using the 2D mask region data structure 932 (block 938). Method 900includes converting the filtered 2D box coordinates into 3D coordinates(block 940). The 3D-guided 2D box detection sub-process 908 outputs adata structures 942 containing 3D boxes found using 2D box detections.

A full scene box detection sub-process 944 receives the 3D box detectiondata structure 932 and the data structures 942 containing 3D boxes foundusing 2D box detections. Method 900 includes transforming 3D boxcoordinates using calibration data (block 946). In one or moreembodiments, distributed processing is performed in order to speedcarton detection, to increase resolution and size of an imaged scene,and to improve detection accuracy. The 3D-guided 2D box detectionsub-process 908 can consolidate the results from each sector or quadrantof the scene. To this end, method 900 includes combining transformed 3Dbox coordinates from each quadrant (block 948).

FIG. 10 illustrates another example method 1000 for 2D, 3D, and3D-guided 2D box detection. In one or more embodiments, method 1000includes receiving RGB-D sensor data by a box detection system (block1002). Method 1000 includes performing 2D RGB process to general finalsegment list (block 1004), such as described in greater detail in FIG.12. With continued reference to FIG. 10, method 1000 includes generating3D point cloud data (block 1006). Method 1000 includes down sampling 3Dpoint cloud data for noise removal (block 1008). Method 1000 includessegmenting the down sampled 3D point cloud data (block 1010). Method1000 includes determining whether there is another segment to filter(decision block 1012). In response to determining that there is anothersegment to filter in decision block 1012, method 1000 includesdetermining whether the selected segment has a size greater than asegment threshold (decision block 1014). In response to determining thatthe selected segment does not have a size greater than a segmentthreshold in decision block 1014, method 1000 includes discarding thesmall segment (block 1016). Method 1000 then returns to decision block1012. In response to determining that die selected segment does not havea size greater than a segment threshold in decision block 1014, method1000 returns to decision block 1012. In response to determining thatthere is not another segment to filter in decision block 1012, method1000 includes filtering large segments out for 2D processing, leavingfiltered 3D results and large segment results (block 1018). Method 1000includes parsing any large segment results into masks (block 1020).Method 1000 includes performing 2D RGB process on the masks (block1022). Method 1000 includes compiling final 2D segment list from RGBimage and masks for each sector or quadrant of detection system intofinal 2D results (block 1024). Method 1000 includes compiling filtered3D results from each sector or quadrant of detection system into final3D results (block 1026). Method 1000 includes combining final 3D and 2Dresults into final results (block 1028). Method 1000 includes passingthe final results to an automation controller for carton loading orunloading operation (block 1030).

FIG. 11 illustrates an example method 1100 for 3D processing performedas part of the method 1000 (FIG. 10). In one or more embodiments, method1100 includes receiving 3D data (block 1102). Method 1100 includesremoving invalid points (block 1104). Method 1100 includes receiving atool rotation (θ) (block 1106). Method 1100 includes rotating cloud (−θ)back (block 1108). Method 1100 includes trimming data according toquadrant of 3D sensor (block 1110). Method 1100 includes performingstatistical outlier removal to remove grains in scene (block 1112).Method 1100 includes performing polynomial smoothing on 3D scene (block1114). Method 1100 includes removing invalid 3D points (block 1116).Method 1100 includes performing region growing segmentation (block1118). Method 1100 includes analyzing clusters of 3D data from regiongrowing segmentation (block 1120). Method 1100 includes determiningwhether cluster length and width are both greater than a clusterthreshold (decision block 1122). In response to determining that thecluster length and width are both not greater than a cluster thresholdin decision block 1122, method 1100 includes fitting plane to cluster(block 1124). Method 1100 includes localizing plane (block 1126).

In response to determining that the cluster length and width are bothgreater than a cluster threshold in decision block 1122, method 1100includes rotating cloud (+θ) back (block 1128). Method 1100 includesapplying extrinsic calibration to find rotation and translation (<R, T>)of cluster to bring to frame of RGB (block 1130). Method 1100 includesapplying intrinsic calibration to cluster (block 1132). Method 1100include drawing a 2D mask (block 1134). Method 1100 includes getting boxlocations from RGB module for 2D mask (block 1136). Method 1100 includesapplying extrinsic and intrinsic calibration to get 3D coordinates fromRGB pixel of box location (block 1138). Method 1100 includes slicingcorresponding regions using pick point and dimension from RGB module(block 1140). Method 1100 includes getting bottom right corner from 3Dcluster (block 1142). After localizing plane in block 1126 or aftergetting bottom right corner from 3D cluster in block 1142, method 1100includes adding bottom right corner coordinates to pick points stack(block 1144). Method 1100 includes (block 1146).

FIG. 12 illustrates an example method 1200 for 2D processing performedas part of the method 1000 (FIG. 10). Beginning with an edge detectionmodule 1202, method 1200 includes receiving 2D RGB image (block 1204).Method 1200 includes performing adaptive thresholding of red layer ofRGB image (block 1206). Method 1200 includes performing adaptivethresholding of green layer of RGB image (block 1208). Method 1200includes performing adaptive thresholding of blue layer of RGB image(block 1210). Method 1200 includes generating first common mask using ORoperation (block 1212). Method 1200 includes perform Difference ofGaussian (DOG) of blue layer of RGB image (block 1214). Method 1200includes generating second common mask from first common mask and DOGmask using OR operation (block 1216). Method 1200 includes performingedge detection using graph models (block 1218). Method 1200 includesgenerating a third common mask from second common mask and edge detectedmask using AND operation (block 1220). Then in a blob detection andsmall segment filtration module 1222, method 1200 includes finalizingedge map (block 1224). Method 1200 includes performing blob detection(block 1226). Method 1200 includes determining whether blob size isgreater than a blob threshold (decision block 1228). In response todetermining that the blob size is not greater than a blob threshold indecision block 1228, method 1200 includes discarding the small blobsegment (block 1230). In response to determining that the blob size isgreater than a blob threshold in decision block 1228, method 1200filtering includes small boxes inside a large box (block 1232). Method1200 includes generating final box list (block 1234).

FIG. 1300 illustrates a method 1300 of combining results of 3D and 2Dbox detection analysis. A carton detector obtains 3D data (block 1302).The carton detector reads an angle from a tool whose position dictatesan angle of a sensor that obtained the 3D data and the carton detectorcounter rotates the 3D data to a normalized position (block 1304).Method 1300 includes the detector performing 3D processing to identifyboxes within the 3D data (block 1306). The result of the detection inblock 1306 is any boxes identified from 3D processing (block 1308). Thecompletion of block 1306 can serve as a trigger for a next iteration of3D data (block 1310). After block 1308, the carton detector extractsblobs that are bigger than a box from the 3D data that require further2D processing in order to detect boxes (block 1312). The blob of 3D datais rotated and transformed from a 3D frame to an RGB frame (block 1314).Method 1300 then includes obtaining RGB data (block 1316). The source ofthe RGD data can be directly from a 2D sensor or can be the rotated andtransformed 3D data. The detector performs 2D processing to identifyboxes (block 1318). The completion of block 1318 can serve as a triggerfor a next iteration of 2D data (block 1320). Method 1300 then includesrotating and transforming the identified 2D boxes to the 3D frame (block1322). The result of the detection in block 1322 can be any boxesidentified from 3D augmented 2D processing (block 1324). Alternativelyor in addition, the result of the detection in block 1322 can be anyboxes identified from 2D processing (block 1326). The identified boxes1308, 1324, 1326 are combined (block 1328). Method 1300 includescommunicated the combined identified boxes to a vehicle/tool controller1330 to perform one or a loading or an unloading procedure with thebenefit of knowing the location of boxes (block 1330). Then method 1300ends.

As used herein, processors may be any programmable microprocessor,microcomputer or multiple processor chip or chips that can be configuredby software instructions (applications) to perform a variety offunctions, including the functions of the various embodiments describedabove. In the various devices, multiple processors may be provided, suchas one processor dedicated to wireless communication functions and oneprocessor dedicated to running other applications. Typically, softwareapplications may be stored in the internal memory before they areaccessed and loaded into the processors. The processors may includeinternal memory sufficient to store the application softwareinstructions. In many devices the internal memory may be a volatile ornonvolatile memory, such as flash memory, or a mixture of both. For thepurposes of this description, a general reference to memory refers tomemory accessible by the processors including internal memory orremovable memory plugged into the various devices and memory within theprocessors.

The foregoing method descriptions and the process flow diagrams areprovided merely as illustrative examples and are not intended to requireor imply that the steps of the various embodiments must be performed inthe order presented. As will be appreciated by one of skill in the artthe order of steps in the foregoing embodiments may be performed in anyorder. Words such as “thereafter,” “then,” “next,” etc. are not intendedto limit the order of the steps; these words are simply used to guidethe reader through the description of the methods. Further, anyreference to claim elements in the singular, for example, using thearticles “a,” “an” or “the” is not to be construed as limiting theelement to the singular.

The various illustrative logical blocks, modules, circuits, andalgorithm steps described in connection with the embodiments disclosedherein may be implemented as electronic hardware, computer software, orcombinations of both. To clearly illustrate this interchangeability ofhardware and software, various illustrative components, blocks, modules,circuits, and steps have been described above generally in terms oftheir functionality. Whether such functionality is implemented ashardware or software depends upon the particular application and designconstraints imposed on the overall system. Skilled artisans mayimplement the described functionality in varying ways for eachparticular application, but such implementation decisions should not beinterpreted as causing a departure from the scope of the presentinvention.

The hardware used to implement the various illustrative logics, logicalblocks, modules, and circuits described in connection with theembodiments disclosed herein may be implemented or performed with ageneral purpose processor, a digital signal processor (DSP), anapplication specific integrated circuit (ASIC), a field programmablegate array (FPGA) or other programmable logic device, discrete gate ortransistor logic, discrete hardware components, or any combinationthereof designed to perform the functions described herein. Ageneral-purpose processor may be a microprocessor, but, in thealternative, the processor may be any conventional processor,controller, microcontroller, or state machine. A processor may also beimplemented as a combination of computing devices, e.g., a combinationof a DSP and a microprocessor, a plurality of microprocessors, one ormore microprocessors in conjunction with a DSP core, or any other suchconfiguration. Alternatively, some steps or methods may be performed bycircuitry that is specific to a given function.

In one or more exemplary embodiments, the functions described may beimplemented in hardware, software, firmware, or any combination thereof.If implemented in software, the functions may be stored on ortransmitted over as one or more instructions or code on a non-transitoryprocessor-readable, computer-readable, or server-readable medium or anon-transitory processor-readable storage medium. The steps of a methodor algorithm disclosed herein may be embodied in a processor-executablesoftware module or processor-executable software instructions which mayreside on a non-transitory computer-readable storage medium, anon-transitory server-readable storage medium, and/or a non-transitoryprocessor-readable storage medium. In various embodiments, suchinstructions may be stored processor-executable instructions or storedprocessor-executable software instructions. Tangible, non-transitorycomputer-readable storage media may be any available media that may beaccessed by a computer. By way of example, and not limitation, suchnon-transitory computer-readable media may comprise RAM, ROM, EEPROM,CD-ROM or other optical disk storage, magnetic disk storage or othermagnetic storage devices, or any other medium that may be used to storedesired program code in the form of instructions or data structures andthat may be accessed by a computer. Disk and disc, as used herein,includes compact disc (CD), laser disc, optical disc, digital versatiledisc (DVD), floppy disk, and Blu-Ray™ disc where disks usually reproducedata magnetically, while discs reproduce data optically with lasers.Combinations of the above should also be included within the scope ofnon-transitory computer-readable media. Additionally, the operations ofa method or algorithm may reside as one or any combination or set ofcodes and/or instructions on a tangible, non-transitoryprocessor-readable storage medium and/or computer-readable medium, whichmay be incorporated into a computer program product.

While the disclosure has been described with reference to exemplaryembodiments, it will be understood by those skilled in the art thatvarious changes may be made and equivalents may be substituted forelements thereof without departing from the scope of the disclosure. Inaddition, many modifications may be made to adapt a particular system,device or component thereof to the teachings of the disclosure withoutdeparting from the essential scope thereof. Therefore, it is intendedthat the disclosure not be limited to the particular embodimentsdisclosed for carrying out this disclosure, but that the disclosure willinclude all embodiments falling within the scope of the appended claims.Moreover, the use of the terms first, second, etc. do not denote anyorder or importance, but rather the terms first, second, etc. are usedto distinguish one element from another.

For clarity, the robotic carton handling system 100 (FIG. 1) isdescribed herein as unloading cartons, which can be corrugated boxes,wooden crates, polymer or resin totes, storage containers, etc. Themanipulator head can further engage articles that are products that areshrink-wrapped together or a unitary product. In one or moreembodiments, aspects of the present innovation can be extended to othertypes of manipulator heads that are particularly suited to certain typesof containers or products. The manipulator head can employ mechanicalgripping devices, electrostatic adhesive surfaces, electromagneticattraction, etc. Aspects of the present innovation can also be employedon a single conventional articulated arm.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the disclosure.As used herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

The description of the present disclosure has been presented forpurposes of illustration and description, but is not intended to beexhaustive or limited to the disclosure in the form disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope of the disclosure. Thedescribed embodiments were chosen and described in order to best explainthe principles of the disclosure and the practical application, and toenable others of ordinary skill in the art to understand the disclosurefor various embodiments with various modifications as are suited to theparticular use contemplated.

What is claimed is:
 1. A method of determining locations of individualcartons in a handling system, the method comprising: receiving, by acarton detection system that is positioned on a robotic carton handlingsystem, a two-dimensional (2D) image and a three dimensional (3D) pointcloud of at least one portion of a carton pile; detecting, by aprocessing subsystem in connection with the carton detection system, aset of segments within the 3D point cloud; detecting, by the processingsubsystem in connection with the carton detection system, another set ofsegments within the 3D point cloud; qualifying, by the processingsystem, the set of segments as 3D detected cartons and the another setof segments as 2D detected cartons; combining, by the processingsubsystem, the 2D and 3D detected cartons into a detection result; andconverting, by the processing subsystem, the detection result usingcalibration information into a 3D location for cartons targeted forremoval.
 2. The method of claim 1, wherein the set of segments arequalified as the 3D detected cartons if the set of segments are lesserthan a segment threshold.
 3. The method of claim 1, wherein the anotherset of segments are qualified as the 2D detected cartons if the anotherset of segments are greater than a segment threshold.
 4. The method ofclaim 3, wherein when the another set of segments are greater than thesegment threshold: determining a 2D mask corresponds to the another setof segments; determining a portion of the 2D image that corresponds tothe 2D mask; detecting the another set of segments within the portion ofthe 2D image; and qualifying detected another set of segments as the 2Ddetected cartons.
 5. The method of claim 1, further comprising:receiving the 2D image and 3D point cloud from one or more sensors todetect the at least one portion of the carton pile; receiving a 2D imageand 3D point cloud from another one or more sensors to detect acontiguous portion of the carton pile; and combining the 2D and 3Ddetected cartons from both the one portion and the contiguous portion toform the detection result.
 6. The method of claim 1, wherein qualifyingthe another set of segments as the 2D detected cartons further comprisesdiscarding, by the processing subsystem, edges that form a smallerrectangle fully encompassed within a larger rectangle.
 7. The method ofclaim 1, further comprising stitching the 2D image and the 3D pointcloud together to obtain an image of the carton pile.
 8. The method ofclaim 1, further comprising. performing one of a loading operation andan unloading operation by the robotic carton handling system using the3D location.
 9. A carton detection system to facilitate unloadingcartons in a carton pile by a robotic carton handling system: a sensorconfigured to provide a two-dimensional (2D) optical image and athree-dimensional (3D) point cloud of at least one portion of a cartonpile resting on a floor of a transportation carrier; a processingsubsystem in communication with the sensor, the processing subsystem:detects a set of segments within the 3D point cloud; detects another setof segments within the 3D point cloud; qualifies the set of segments as3D detected cartons and the another set of segments as 2D detectedcartons; combine the 2D and 3D detected cartons into a detection result;and convert the detection result using calibration information into a 3Dlocation for cartons targeted for removal to enable the robotic cartonhandling system to remove the cartons from the carton pile.
 10. Therobotic carton handling system of claim 9, wherein the set of segmentsare qualified as the 3D detected cartons if the processing subsystemdetermines that the set of segments are lesser than a segment threshold.11. The robotic carton handling system of claim 9, wherein the anotherset of segments are qualified as the 2D detected cartons if the if theprocessing subsystem determines that the another set of segments aregreater than a segment threshold.
 12. The robotic carton handling systemof claim 11, wherein in response to determining that the another set ofsegments are greater than the segment threshold, the processingsubsystem: determines a 2D mask that corresponds to the another set ofsegments; determines a portion of the 2D image that corresponds to the2D mask; detects the another set of segments within the portion of the2D image; and qualifies detected another set of segments as the 2Ddetected cartons.
 13. The robotic carton handling system of claim 9,wherein: the sensor is positioned on the robotic carton handling systemto detect the one portion of the carton pile; the robotic cartonhandling system further comprises another sensors positioned on therobotic carton handling system to detect a contiguous portion of thecarton pile; and the processing subsystem combines the 2D and 3Ddetected cartons from both the one portion and the contiguous portion toform the detection result.
 14. The robotic carton handling system ofclaim 9, wherein the processing subsystem qualifies the one or moresegments as the 2D detected cartons at least in part by discarding edgesthat form a smaller rectangle fully encompassed within a largerrectangle.
 15. The robotic carton handling system of claim 9, whereinthe processing subsystem is further configured to stitch the 2D imageand the 3D point cloud together to obtain an image of the carton pile.16. The robotic carton handling system of claim 9, further comprising anautomation controller in communication with the processing subsystem,the automation controller causes a robotic carton manipulator to performone of the loading operation and the unloading operation by the roboticcarton handling system using the 3D location.
 17. A material handlingsystem comprising: a robotic carton handling system for unloadingcartons in a carton pile, the robotic carton handling system movableacross a floor, the robotic carton handling system comprising: a mobilebody; a movable robotic manipulator attached to the mobile body andcomprising an end effector at an end thereof, the end effectorconfigured to unload one or more cartons from the carton pile; aconveyor mounted on the mobile body configured to receive the one ormore cartons from the end effector and to move the one or more cartonstowards a rear of the robotic carton handling system; a carton detectionsystem comprising: one or more sensors coupled respectively to one ofthe mobile body and the movable robotic manipulator to provide atwo-dimensional (2D) optical image and a three-dimensional (3D) pointcloud of at least one portion of carton pile resting on a floor of atransportation carrier; a processing subsystem in communication with theone or more sensors, the processing subsystem: detects a set of segmentswithin the 3D point cloud; detects another set of segments within the 3Dpoint cloud; qualifies the set of segments as 3D detected cartons andthe another set of segments as 2D detected cartons; combine the 2D and3D detected canons into a detection result; and convert the detectionresult using calibration information into a 3D location for cartonstargeted for removal, an automation controller in communication with theprocessing subsystem, the automation controller causes a robotic cartonmanipulator to perform the selected one of the loading operation and theunloading operation by the robotic carton handling system using the 3Dlocation; and an extendable conveyor system having a proximal endcoupled to a stationary conveyor system and a movable distal endpositioned proximate to the robotic carton handling system to transfercartons between the stationary conveyor and the robotic carton handlingsystem.
 18. The material handling system of claim 17, wherein theprocessing subsystem is further configured to: receive the 2D image and3D point cloud from one or more sensors positioned on the robotic cartonhandling system to detect the one portion of the carton pile; receive a2D image and 3D point cloud from another one or more sensors positionedon the robotic carton handling system to detect a contiguous portion ofthe carton pile; and combine the 2D and 3D detected cartons from boththe one portion and the contiguous portion to form the detection result.19. The material handling system of claim 17, where in the processingsubsystem is further configured to: determine a 2D mask that correspondsto the another set of segments; determine a portion of the 2D image thatcorresponds to the 2D mask; detect the another set of segments withinthe portion of the 2D image; and quality detected another set ofsegments as the 2D detected cartons.